Persistent Queue
Cedalo Persistent Queue for Eclipse Mosquitto
In several use cases of an MQTT Message Broker the broker has to deal with unstable or unreliable connectivity. Sometimes clients may get disconnected quite frequently and/or reconnect after a significant amount of time. This very often happens with moving assets, which have an unreliable mobile network connectivity. Or especially in federated broker setups some broker maybe offline for a longer period of time. The Cedalo Persistent Queue feature will help to deal with such scenarios when larger amount of messages need to be stored for either the persistent clients being offline or broker bridge connection not being able to send out data.
Memory message queue
The Eclipse Mosquitto broker supports the queuing of messages for clients that are persistently connected but temporarily offline. This functionality extends to outgoing MQTT bridge connections, holding messages in a queue until the target broker becomes available. These queued messages are stored in memory for all persistent connections that are offline. To ensure data integrity, messages are also backed up in a persistence store through the use of a persistence plugin, such as the Persist SQLite plugin. However, the capacity for queued messages is inherently constrained by the broker's available main memory. This limitation can become apparent in scenarios with high message volumes or prolonged network outages.
In Pro Mosquitto it is possible to avoid a crash of the broker due to memory exhaustion. The user may limit the number of messages, which will be queued for a single client or a single outgoing bridge connection by using the config entry max_queued_messages.
If the broker is stopped and restarted with queued messages for one of multiple connections all messages will be restored from the persistence store before the broker will start its operation.
Persistent message queue
The introduction of the Cedalo Persistent Queue feature liberates users from the constraints of main memory for storing messages for offline connections. By configuring the max_persist_queued_messages setting, users can specify the number of messages to be queued on disk per client, enabling a reduction in the max_queued_messages setting and consequently, the memory footprint.
To leverage the persistent queue functionality, a compatible persistence plugin must be configured. Initially, only the Persistent SQLite store supports the Cedalo Persistent Queue feature.
Upon restarting, if the broker faces a large volume of queued messages for one or more clients, it will load only the necessary messages into memory and keep the remainder on disk. This approach significantly enhances the restart process for brokers managing extensive message queues.
The availability and capacity of the persistent queue may vary based on your license. License details, including features and expiration dates, can be found in the broker log, indicated by entries such as:
License: Feature persist-queued-messages: 1000000 AVAILABLE until 2024-05-02T15:53:46.